WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




PCT 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 6 : 
H04N 1/32 



Al 



(11) International Publication Number: WO 99/57885 

(43) International Publication Date: 1 1 November 1999 (1 1 .1 1 .99) 



(21) International Application Number: PCT7EP99/02928 

(22) International Filing Date: 29 April 1999 (29.04.99) 



(30) Priority Data: 

09/070,524 



30 April 1998 (30.04.98) 



US 



(71) Applicants: MEDIASEC TECHNOLOGIES LLC [US/US]; 
Suite % 321 S. Main Street, Providence, RI 02903 (US). 
FRAUNHOFER CENTER FOR RESEARCH IN COM- 
PUTER GRAPHICS, INC. [US/US]; Suite 2, 321 S. Main 
Street, Providence, RI 02903 (US). 

(71) (72) Applicant and Inventor: KOCH, Eckhard [DE/DE]; Sun- 

dernholz 71, D-45134 Essen (DE). 

(72) Inventor: ZHAO, Jian; 64 Thomas Olney Common, Provi- 

dence, RI 02904 (US). 

(74) Agent: ROSLER, Uwe; Landsberger Strasse 480 a, D-81241 
Miinchen (DE). 



(81) Designated States: JP, KR f European patent (AT, BE, CH, CY, 
DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, NL, PT, 
SE). 



Published 

With international search report. 



(54) Title: DIGITAL AUTHENTICATION WITH ANALOG DOCUMENTS 
(57) Abstract 

Techniques for protecting the security of digital representations, and of analog forms made from them. The techniques include 
authentication techniques that can authenticate both a digital representation and an analog form produced from the digital representation, an 
active watermark that contains program code that may be executed when the watermark is read, and a watermark agent that reads watermarks 
and sends messages with information concerning the digital representations that contain the watermarks. The authentication techniques use 
semantic information to produce authentication information. Both the semantic information and the authentication information survive when 
an analog form is produced from the digital representation. In one embodiment, the semantic information is alphanumeric characters and 
the authentication information is either contained in a watermark embedded in the digital representation or expressed as a bar code. With 
the active watermark, the watermark includes program code. When a watermark reader reads the watermark, it may cause the program code 
to be executed. One application of active watermarks is making documents that send messages when they are operated on. A watermark 
agent may be either a permanent resident of a node in a network or of a device such as a copier or it may move from one network node to 
another. In the device or node, the watermark agent executes code which examines digital representations residing in the node or device 
for watermarked digital representations that are of interest to the watermark agent. The watermark agent then sends messages which report 
the results of its examination of the digital representations. If the watermarks are active, the agent and the active watermark may cooperate 
and the agent may cause some or all of the code that an active watermark contains to be executed. 
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Digital Authentication with Analog Documents 
Cross Reference to Related Patent Applications 

This application has the same Detailed Description as Jian Zhao, Active Watermarks and 
Watermark Agents, assigned to Fraunhofer CRCG and filed on even date with this 
application. 

Background of the Invention 

1. Field of the Invention 

The invention relates generally to digital representations of images and other information and 
more specifically to techniques for protecting the security of digital representations and of 
analog forms produced from them. 

2. Description of the Prior Art 

Nowadays, the easiest way to work with pictures or sounds is often to make digital 
representations of them. Once the digital representation is made, anyone with a computer can 
copy the digital representation without degradation, can manipulate it, and can send it 
virtually instantaneously to anywhere in the world. The Internet, finally, has made it possible 
for anyone to distribute any digital .representation from anywhere in the world 

From the point of view of the owners of the digital representations, there is one problem with 
all of this: pirates, too, have computers, and they can use them to copy, manipulate, and 
distribute digital representations as easily as the legitimate owners and users can. If the 
owners of the original digital representations are to be properly compensated for making or 
publishing them, the digital representations must be protected from pirates. There are a 
number of different approaches that can be used: 

the digital representation may be rendered unreadable except by its intended 

recipients; this is done with encryption techniques; 
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the digital representation may be marked to indicate its authenticity; this is done with 
digital signatures; 

the digital representation may contain information from which it may be determined 
whether it has been tampered with in transit; this information is termed a digest and 
the digital signature often includes a digest; 

the digital representation may contain a watermark, an invisible indication of 
ownership which cannot be removed from the digital representation and may even be 
detected in an analog copy made from the digital representation; and 
the above techniques can be employed in systems that not only protect the digital 
representations, but also meter their use and/or detect illegal use. 
For an example of a system that uses encryption to protect digital representations, see U.S. 
Patent 5,646.999, Saito, Data Copyright Management Method, issued July 8, 1997; for a 
general discussion of digital watermarking, see Jian Zhao, "Look, It's Not There", in: BYTE 
Magazine. January, 1997. Detailed discussions of particular techniques for digital 
watermarking may be found in E. Koch and J. Zhao. "Towards Robust and Hidden Image 
Copyright Labeling", in: Proc. Of 1995 IEEE Workshop on Nonlinear Signal and Image 
Processing. June 20-22, 1995, and in U.S. Pat. No. 5.710,834, Rhoads, Method and 
Apparatus Responsive to a Code Signal Conveyed through a Graphic Image, issued Jan. 20. 
1998. For an example of a commercial watermarking system that uses the digital 
watermarking techniques disclosed in the Rhoads patent, see Digimarc Watermarking Guide. 
Digimarc Corporation, 1997, available at in March, 1998 at 
http : //www . digimarc . com. 

FIG. 1 shows a prior-art system 101 which employs the above protection techniques. A 
number of digital representation clients 105, of which only one. digital representation client 
1050) is shown, are connected via a network 103 such as the Internet to a digital 
representation server 129 which receives digital representations from clients 105 and 
distributes them to clients 105. Server 1 29 includes a data storage device 133 which contains 
copied digital representations 135 for distribution and a management data base 139. Server 
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129 further includes a program for managing the digital representations 135. a program for 
reading and writing watermarks 109, a program for authenticating a digital representation and 
confirming that a digital representation is authentic 1 1 1, and a program for encrypting and 
decrypting digital representations 113. Programs 109, 111, and 113 together make up 
security programs 107. 

Client 105 has its own versions of security programs 107; it further has editor/viewer program 
115 which lets the user of client 105 edit and/or view digital representations that it receives 
via network 103 or that are stored in storage device 117. Storage device 117 as shown 
contains an original digital representation 1 1 9 which was made by a user of client 1 05 and a 
copied digital representation 121 that was received from DR Server 129. Of course, the user 
may have made original representation 1 19 by modifying a copied digital representation 
Editor/viewer program 115, finally, permits the user to output digital representations to 
analog output devices 123. Included among these devices are a display 123, upon which an 
analog image 124 made from a digital representauon may be displayed and a printer 127 upon 
which an analog image 126 made from the digital representation may be printed A 
loudspeaker may also be included in analog output devices 123. The output of the analog 
output device will be termed herein an analog form of the digital representation. For 
example, if the output device is a printer, the analog form is printed sheet 126; if it is a display 
device, it is display 124. 

When client 1050) wishes to receive a digital representation from server 129, it sends a 
message requesting the the digital representation to server 129. The message includes at 
least an identification of the desired digital representation and an identification of the user 
Manager 131 responds to the request by locating the digital representation in CDRs 135 
consulting management data base 139 to determine the conditions under which the digital 
representation may be distributed and the status of the user of client 105 as a customer If 
the information in data base 139 indicates to manager 131 that the transaction should go 
forward, manager 131 sends client 105© a copy of the selected digital representation In the 
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course of sending the copy, manager 131 may use watermark reader/writer 109 to add a 
watermark to the digital representation, use authenticator/confirmer 1 11 to add authentication 
information, and encrypter/decrypter 113 to encrypt the digital representation in such a 
fashion that it can only be decrypted in DR client 105(j). 

When client 105© receives the digital representation, it decrypts it using program 113, 
confirms that the digital representation is authentic using program 1 1 1, and editor/viewer 
1 15 may use program 109 to display the watermark. The user of client 105(j) may save the 
encrypted or unencrypted digital representation in storage \ 1 7. The user of client 1050) may 
finally employ editor/viewer 1 15 to decode the digital representation and output the results 
of the decoding to an analog output device 123. Analog output device 123 may be a display 
device 125, a printer 127, or in the case of digital representations of audio, a loudspeaker. 



It should be pointed out that when the digital representation is displayed or printed in analog 
form, the only remaining protection against copying is watermark 128, which cannot be 
perceived in the analog form by the human observer, but which can be detected by scanning 
the analog form and using a computer to find watermark 1 28. Watermark 1 28 thus provides 
a backup to encryption: if a digital representation is pirated, either because someone has 
broken the encryption, or more likely because someone with legitimate access to the digital 
representation has made illegitimate copies, the watermark at least makes it possible to 
determine the owner of the original digital representation and given that evidence, to pursue 
the pirate for copyright infringement and/or violation of a confidentiality agreement. 

If the user of client 105© wishes to send an original digital representation 1 19 to DR server 
129 for distribution, editor/viewer 1 15 will send digital representation 1 19 to server 129 In 
so doing, editor/viewer 115 may use security programs 107 to watermark the digital 
representation, authenticate it. and encrypt it so that it can be decrypted only by DR Server 
129. Manager 131 in DR server 129 will, when it receives digital representation 1 19 use 
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security programs 107 to decrypt digital representation 1 19, confirm its authenticity, enter 
information about it in management data base 139, and store it in storage 133 

In the case of the Digimarc system referred to above, manager 13 1 also includes a World 
Wtde Web spider, that is, a program that systematically follows World Wide Web links such 
as HTTP and FTP links and fetches the material pointed to by the links. 

Manager program 13 1 uses watermark reading/writing program to read any watermark, and 
if the watermark is known to management database .139. manager program 131 takes 
whatever action may be required, for example, determining whether the site from which the 
digital representation was obtained has the right to have it. and if not, notifying the owner of 
the digital representation. 

While encryption, authentication, and watermarking have made it much easier for owners of 
digital representations to protect their property, problems still remain. One such problem is 
that the techniques presently used to authenticate digital documents do not work with analog 
forms; consequently, when the digital representation is output in analog form, the 
authentication is lost. .Another is that present-day systems for managing di B ital 
representations are not flexible enough. A third is that watermark checking such as that done 
by the watermark spider described above is limited to digital representations available on the 
Internet. It is an object of the present invention to overcome the above problems and thereby 
to provide improved techniques for distributing digital representations. 

Summary of the Invention 

The problem that digital authentication techniques are limited to digital representations is 
overcome by an authentication technique that is based on semantic information, that is 
■"formation that must be present in any analog form made from the digital representation 
The semantic information is used to produce identification information such as a digest and 
the d,gest is added to the digital representation in a manner that does not affect the semantic 
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information. In one embodiment, the identification information is embedded in the digital 
representation as a watermark; in another, the digest is expressed as a barcode. When a 
digital representation or analog form contains authentication information that is based on the 
semantic information, the representation or form is authenticated by again using the semantic 
information to compute authentication information and then comparing the newly-computed 
authentication information with the authentication information in the representation or form. 
If the two match, the digital representation or analog form is authentic. Depending on the 
semantic information and the purpose of the authentication, the match may either be precise 
or fuzzy. Among the uses of authentication based on semantic information are authentication 
of digital forms of electronic documents, authentication of paper digital cash, authentication 
of paper digital checks, and authentication of identification cards such as bankcards. 

Other objects and advantages of the invention will be apparent to those skilled in the ans to 
which the invention pertains upon perusing the following Detailed Description and Drawing, 
15 wherein: 
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Brief Description of the Drawing 

FIG. 1 is a block diagram of a prior-art system for securely distributing digital 
representations; 

FIG. 2 is a diagram of a first embodiment of an analog form that can be authenticated; 
FIG. 3 is a diagram of a second embodiment of an analog form that can be authenticated: 
FIG. 4 is a diagram of a system for adding authentication information to an analog form; 
FIG. 5 is a diagram of a system for authenticating an analog form; 
FIG. 6 is a diagram of a system for making an active watermark; 
FIG. 7 is an example of code from an active watermark; 

FIG. 8 is a diagram of a system for executing the code in an active watermark; 
FIG. 9 is a diagram of a system for producing a watermark agent; 
FIG. 10 is a diagram of a system for receiving a watermark agent; 
FIG. 11 is a detailed diagram of access information 603; and 
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FIG. 12 is an example of code executed by a watermark agent. 

The reference numbers in the drawings have at least three digits. The two rightmost digits 
are reference numbers within a figure; the digits to the left of those digits are the number of 
the figure in which the item identified by the reference number first appears. For example, an 
item with reference number 203 first appears in FIG. 2. 

Detailed Description 

The following Detailed Description will first disclose a technique for authenticating digital 
representations that survives output of an analog form of the digital representation, will then 
disclose active watermarks, that is, watermarks that contain programs, and will finally 
disclose watermark agents, that is, programs which examine the digital watermarks on digital 
representations stored in a system and thereby locate digital representations that are being 
used improperly. 

Authentication that is preserved in analog forms: FIGs. 2-5 

Digital representations are authenticated to make sure that they have not been altered in 
transit. Alteration can occur as a result of transmission errors that occur during the course 
of transmission from the source of the digital representation to its destination, as a result of 
errors that arise due to damage to the storage device being used to transport the digital 
representation, as a result of errors that arise in the course of writing the digital 
representation to the storage device or reading the digital representation from the storage 
device, or as a result of human intervention. A standard technique for authentication is to 
make z digest of the digital representation and send the digest to the destination together with 
the digital representation. At the destination, another digest is made from the digital 
representation as received and compared with the first. If they are the same, the digital 
representation has not changed. The digest is simply a value which is much shorter than the 
digital representation but is related to it such that any change in the digital representation will 
with very high probability result in a change to the digest. 
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Where human intervention is a serious concern, the digest is made using a oneway hash 
function, that is, a function that produces a digest from which it is extremely difficult or 
impossible to learn anything about the input that produced it. The digest may additionally 
be encrypted so that only the recipient of the digital representation can read it. A common 
technique is to use the encrypted digest as the digital signature for the digital representation, 
that is, not only to show that the digital representation has not been altered in transit, but also 
to show that it is from whom it purports to be from. If the sender and the recipient have 
exchanged public keys, the sender can make the digital signature by encrypting the digest with 
the sender's private key. The recipient can use the sender's public key to decrypt the digest, 
and having done that, the recipient compares the digest with the digest made from the 
received digital representation. If they are not the same, either the digital representation has 
been altered or the digital representation is not from the person to whom the public kev used 
to decrypt the digest belongs. For details on authentication, see Section 3.2 of Bruce 
Schneier, Applied Cryptography, John Wiley and Sons. 1994. 

The only problem with authentication is that it is based entirely on the digital representation 
The information used to make the digest is lost when the digital representation is output in 
analog form. For example, if the digital representation is a document, there is no way of 
determining from a paper copy made from the digital representation whether the digital 
representation from which the paper copy was made is authentic or whether the paper copy 
is itself a true copy of the digital representation. 

While digital watermarks survive and remain detectable when a digital representation is 
output in analog form, the authentication problem cannot be solved simply by embedding the 
digest or digital signature in the watermark. There are two reasons for this: 

Watermarking changes the digital representation; consequently, if a digital 
representation is watermarked after the original digest is made, the watermarking 
invalidates the original digest, i.e.. it is no longer comparable with the new digest that 
the recipient makes from the watermarked document. 
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More troublesome still, when a digital representation is output in analog form, so 
much information about the digital representation is lost that the digital representation 
cannot be reconstructed from the analog form. Thus, even if the original digest is still 
valid, there is no way of producing a comparable new digest from the analog form. 

What is needed to overcome these problems is an authentication technique which uses 
information for authentication which is independent of the particular form of the digital 
representation and which will be included in the analog form when the analog form is output. 
As will be explained in more detail in the following, the first requirement is met by selecting 
semantic information from the digital representation and using only the semantic information 
to make the digest. The second requirement is met by incorporating the digest into the digital 
representation in a fashion such that it on the one hand does not affect the semantic 
information used to make the digest and on the other hand survives in the analog form. In the 
case of documents, an authentication technique which meets these requirements can be used 
not only to authenticate analog forms of documents that exist primarily in digital form, but 
also to authenticate documents that exist primarily or only in analog form, for example paper 
checks and identification cards. 

Semantic information 

The semantic information in a digital representation is that portion of the information in the 
digital representation that must be present in the analog form made from the digital 
representation if the human who perceive the analog form is to consider it a copy of the 
original from which the digital representation was made. For example, the semantic 
information in a digital representation of an image of a document is the representations of the 
alphanumeric characters in the document, where alphanumeric is understood to include 
representations of any kind of written characters or punctuation marks, including those 
belonging to non-Latin alphabets, to syllabic writing systems, and to ideographic writing 
systems. Given the alphanumeric characters, the human recipient of the analog form can 
determine whether a document is a copy of the original, even though the characters mav have 
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different fonts and may have been formatted differently in the original document. There is 
analogous semantic information in digital representations of pictures and of audio information. 
In the case of pictures, it is the information that is required for the human that perceives the 
analog form to agree that the analog form is a copy (albeit a bad one) of the original picture, 
and the same is the case with audio information. 

In the case of a document written in English, the semantic information in the document is the 
letters and punctuation of the document. If the document is in digital form, it may be 
represented either as a digital image or in a text representation language such as those used 
for word processing or printing. In the first case, optical character recognition (OCR) 
technology may be applied to the image to obtain the letters and punctuation; in the second 
case, the digital representation may be parsed for the codes that are used to represent the 
letters and punctuation in the text representation language. If the document is in analog form, 
it may be scanned to produce a digital image and the OCR technology applied to the digital 
image produced by scanning. 

Using semantic information to authenticate an analog form: FIGs. 2 and 3 

Because the semantic information must be present in the analog form, it may be read from the 
analog form and used to compute a new digest. If the old digest was similarly made from the 
semantic information in the digital representation and the old digest is readable from the 
analog form, the new digest and the old digest can be compared as described in the discussion 
of authentication above to determine the authenticity of the analog form. 

FIG. 2 shows one technique 201 for incorporating the old digest into an analog form 203 
Analog form 203 of course includes semantic information 205; here, analog form 203 is a 
Printed or faxed document and semantic information 205 is pan or ail of the alphanumeric 
characters on analog form 203. Sometime before analog form 203 was produced, semantic 
mformauon 205 in the digital representation from which analog form 203 was produced was 
used to make semantic digest 207, which was incorporated into analog form 203 at a location 
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which did not contain semantic information 205 when analog form 203 was printed. In 
some embodiments, semantic digest 207 may be added to the original digital representation; 
in others, it may be added just prior to production of the analog form. Any representation of 
semantic digest 207 which is detectable from analog form 203 may be employed; in technique 
201, semantic digest 207 is a visible bar code. Of course, semantic digest 207 mav include 
additional information; for example, it may be encrypted as described above and semantic 
d!gest 207 may include an identifier for the user whose public key is required to decrypt 
semantic digest 207. In such a case, semantic digest 207 is a digital signature that persist, in 
the analog form. 

With watermarking, the semantic digest can be invisibly added to the analog form. This is 
shown in HG. 3. In technique 301, analog form 303 again includes semantic information 305 
Prior to producing analog form 303, the semantic information in the digital representation 
from which analog form 303 is produced is used as described above to produce semantic 
d.gest 207; this time, however, semantic digest 207 is incorporated into watermark 307 
which is added to the digital representation before the analog form is produced from the 
d.gual representation and which, like the bar code of FIG. 2, survives production of the 
analog form. A watermark reader can read watermark 307 from a digital image made by 
scanning analog form 303, and can thereby recover semantic digest 207 from watermark 307 
As was the case with the visible semantic digest, the semantic digest in watermark 307 may 
be encrypted and may also function as a digital signature. 

Adding a semantic digest to an analog form: FIG. 4 

FIG. 4 shows a system 401 for adding a semantic digest to an analog form 203 . The process 
begms with digital representation 403. whose contents include semantic information ^05 
D.gttal representation 403 is received by semantics reader 405. which reads semantic 
mformation 205 from digital representation 403. Semantics reader 405's operation will 
depend on the form of the semantic information. For example, if digital representation 403 
represents a document, the form of the semantic information will depend on how the 
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document is represented. If it „ represented as a bit-map image, the semanuc information will 
be images of alphanumeric characters in the bit map; if it is represented using one of the manv 
representations of documents that express alphanumeric characters as codes, the semantic 
.nformation will be the codes for the alphanumeric characters. In the firs: case, semantics 
reader 405 will be an optical character reading (OCR) device; in the second, it will simolv 
parse the document representation looking for character codes. 

In any case, at the end of the process, semantics reader 405 will have extracted some form 
of semantic information, for example the ASCII codes corresponding to the alphanumeric 
charaaers, from representation 403. This digital information is then provided to dieest maker 
409, which uses it to make semantic digest 41, in any of many known wavs. Depending on 
the kind of document the semantic digest is made from and its intended use, the semantic 
digest may have a form which requires an exact match with the new dieest or mav have a 
form which permits a "fuzzy" match. Digital representation 403 and semantic digest^ 1 are 
then provided to digest incorporator 413, which incorporates a representation 207 of digest 
41 1 into the digital representation used to produce analog form 203. As indicated above the 
representation must be incorporated in such a way that it does not affect semantic information 
205. Incorporator 413 then outputs the representation it produces to analog form producer 
415, whtch produces analog form 203 in the usual fashion. Analog form 203 of course 
includes semantic information 205 and representation 207 of semantic digest 4,1 Here the 
bar code is used, but representation 207 could equaHv be pan of a watermark, as in analog 
form 303. Components 405, 409. and 4,3 may be implemented as programs executed on a 
digual computer system; analog form producer 4 1 5 may be any device which can output an 
analog form. 

Authenticating an analog form that has a semantic digest 

FIG. 5 shows a system 501 for authenticating an analog form 503 that has a semantic digest 
-07. Analog form 503 is first provided to semantic digest reader 505 and to semantics reader 
>05. Semantic digest reader 505 reads semantic digest 207; if semantic digest 207 is a bar 
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code, semantic digest reader 505 is a bar code reader; if semantic digest 207 is included in a 
digital watermark, semantic digest reader 505 is a digital watermark reader which receives 
its input from a scanner. If semantic digest 505 must be decrypted, semantic digest reader 
505 will do that as well. In some cases, that may require sending the encrypted semantic 
digest to a remote location that has the proper key 

Semantics reader 507 reads semantic information 305. If analog form 503 is a document, 
semantics reader 507 is a scanner which provides its output to OCR software. With other 
images, the scanner provides its output to whatever image analysis software is required to 
analyze the features of the image that make up semantic information 305. If analog form 503 
is audio, the audio will be input to audio analysis software. Once the semantics information 
has been reduced to semantics data 509, it is provided to semantic digest maker 511, which 
makes a new semantic digest 513 out of the information. To do so, it uses the same technique 
that was used to make old semantic digest 515 Comparator 5 1 7 then compares old semantic 
digest 515 with new semantic digest 513; if the digests match, comparison result 519 
indicates that analog form 203 is authentic; if they do not. result 5 1 9 indicates that thev are 
not authentic. What "match" means in this context will be explained in more detail below. 

"Matching" semantic digests 

With the digests that are normally used to authenticate digital representations, exact matches 
between the old and new digests are required. One reason for this is that in most dieital 
contexts, "approximately correct" data is useless; another is that the one-way hashes normally 
used for digests are "cryptographic-, that is, the value of the digest reveals nothing about the 
value from which it was made by the hash function, or in more practical terms, a change of 
a smgle bit in the digital representation may result in a large change in the value produced by 
the hash function. Since that is the case, the only comparison that can be made between 
digests is one of equality. 

In the context of authenticating analog forms, the requirement that digests be equal causes 



WO 99/57885 PCT/EP99/02928 

difficulties. The reason for this is that reading semantic information from an analog form is 
an error-prone operation. For example, after many years of effort, OCR technology has 
gotten to the point where it can in general recognize characters with 98% accuracv Jhen it 
begins with a clean copy of a document that is simply formatted and uses a reasonable tvpe 
font. Such an error rate is perfectly adequate for many purposes; but for semantic information 
of any s.ze, a new digest will almost never be equal to the old digest when the new digest 
>s made from semantics data that is 9S% the same as the semantics data that was used to 
make the old semantic digest. On the other hand, if the semantics data obtained from the 
analog form is 9S% the same as the semantics data obtained from the digital representation 
there >s a very high probability that the analog form is in fact an authentic copy of the digital 
representation. 

Precise matches 

Of course, if the semantic information is limited in size and tightly constrained, it may be 
poss.ble to require that the digests be exactly equal. For example, many errors can be 
eliminated if what is being read is specific fields, for example in a check or identification card 
and the OCR equipment is programmed to take the nature of the field's contents into account 
For example, if a field contains only numeric characters, the OCR equipment can be 
programmed to treat the letters o and O as the number 0 and the letters l,i, or I as the number 
1- Moreover, if a match fails and the semantic information contains a character that is eas.ly 
confused by the OCR equipment, the character may be replaced by one of the characters with 
whtch it is confused, the digest may be recomputed, and the match may again be attempted 
with the recomputed digest. 

Fuzzy matches 

Where the semantic information is not tightly constrained, the digests must be made in such 
a fash.on that closely-similar semantic information produces closely-similar digests When 
that ,s the case, matching becomes a matter of determining whether the difference between 
the dl gests is within a threshold value, not of determining whether they are equal. A paper 
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by Marc Schneider and Shih-Fu Chang, "A Robust Content Based Digital Signature for Image 
Authentication", in: Proceedings of the 1996 International Conference on Image Processtng, 
presents some techniques for dealing with related difficulties in the area of digital imaging. 
There, the problems are not caused by loss of information when a digital representation is 
used to make an analog form and by mistakes made in reading analog forms, but rather by 
"lossy" compression of images, that is, compression using techniques which result in the loss 
of information. Because the lost information is missing from the compressed digital 
representation, a digest made using cryptographic techniques from the compressed digital 
representation will not be equal to one made from the digital representation prior to 
compression, even though the compressed and uncompressed representations contain the 
same semantic information. Speaking generally, the techniques presented in the Schneider 
paper deal with this problem by calculating the digest value from characteristics of the image 
that are not affected by compression, such as the spatial location of its features. Where there 
are sequences of images, the digest value is calculated using the order of the images in the 
sequences. 



1. 



Analogous approaches may be used to compute the semantic digest used to authenticate an 
analog form. For example, a semantic digest for a document can be computed like this: 
Set the current length of a digest string that will hold the semantic digest to "0"; 
Starting with the first alphanumeric character in the document, perform" the following 
steps until there are no more characters in the document: 

a. Select a next group of characters; 

b. For the selected group. 

i. replace characters in the group such as O,0,o; I.i.l, 1 ; or c,e that cause 
large numbers of OCR errors with a "don't care" character; 

ii. make a hash value from the characters in the group; 
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i". append the hash value to the semantic digest string, 
c. return to step (a). 

3. When there are no more characters in the document, make the semantic digest from 
the digest string. 

When computed in this fashion, the sequence of values in the semantic digest strine reelects 
the order of the characters in each of the sequences used to compute the dieest. If the 
sequence of values in the new semantic digest that is computed from the analog form has a 
high percentage of matches with the sequence of values in the old semantic digest, there is a 
high probability that the documents contain the same semantic information. 

Applications of authentication with analog forms 

One area of application is authenticating written documents generally. To the extent that the 
document is of any length and the digest is computed from a significant amount of the 
contents, the digest will have to be computed in a fashion which allows fuzzv matching If 
the digest is computed from closely-constrained fields of the document, exact matchina may 
be employed. 



Another area of application is authenticating financial documents such as electronic cash 
electronic checks, and bank cards. Here, the fields from which the digest is computed are 
tightly constrained and an exact match may be required for security. In all of these 
applications, the digest or even the semantic information itself would be encrypted 
described above to produce a digital signature. 



as 



Universal Paper & Digital Cash 

Digital cash is at present a purely electronic medium of payment. A given item of dirital cash 
conststs of a unique serial number and a digital signature. Authentication using semantic 
.nformauon permits digital cash to be printed as digital paper cash. The paper cash is printed 
from an electronic image which has a background image, a serial number, and a money 
amount. The serial number and the money amount are the semantic information. The serial 
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number and the money amount are used to make a digital signature and the digital signature 
is embedded as an electronic watermark into the background image. The paper cash can be 
printed by any machine which needs to dispense money. Thus, an ATM. can dispense digital 
paper cash instead of paper money. Similarly, a vending machine can make chanee with 
digital paper cash and a merchant can do the same. The digital paper cash can be used in the 
same way as paper money. When a merchant (or a vending machine) receives the digital 
paper cash in payment, he or she uses a special scanner (including OCR technology and a 
watermark reader) to detect the watermark (i.e. the serial number and money amount) from 
the printed image, and send them to the bank for verification in the same fashion as is 
presently done with credit cards. 

Digital Checks 

Digital checks can be made using the same techniques as are used for digital paper cash The 
digital check includes a background image, an identifier for the bank account, an amount to 
be paid, and the name of the payer. The payer's private key is used to make a digital 
suture from at least the identification of the bank and the amount to be paid, and the digital 
signature is embedded as an electronic watermark in the background image. Writing a dieital 
check is a three-step process: enter the amount, produce the digital signature from the bank 
account number and the amount using the payer's private key, and embed the digital sienature 
mto the background image. The bank verifies the check by detecting the watermark from the 
digital check., decrypting the digital signature with the payer's public key, and comparing the 
bank account number and the amount from the image with the bank account number and the 
amount on the face of the check. A digital check can be used in either electronic form or 
paper form. In the tatter case, a scanner (including OCR technology and watermark reader) 
■s needed to read the watermark from the paper check. 

Authentication of identification cards 

The techniques described above for authenticating digital paper cash or digital checks can be 
used w, th .denufication cards, including bankcards. The card number or other identification 
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information appears on the face of the card, is encrypted into a digital signature, and is 
embedded as a digital watermark in the background image of the bankcard. The encryption 
can be done with the private key of the institution that .ssues the card. The merchant uses use a 
scanner to detect the digital signature (i.e. card number or other ID) from the card, and compare 
the S] gnature with the authentication stored ins.de the card. This technique can of course be 
combined with conventional authentication techniques such as the holographic logo. 



Active watermarks: FIGs. 6-8 

Heretofore, digital watermarks have been nothing more than labels. They have typically 
contained information such as identifiers for the owner and creator of the digital 
representation and access control information, for example, whether the digital representation 
may be copied or changed. Any kind of information can, however be placed in a digital 
watermark. If the information in the watermark describes an action to be taken the 
watermark becomes active, and the digitaJ representation that contains the active watermark 
becomes active as well This is the reverse of the usual practice of encapsulating a digital 
represents in a program, as is done for example with Microsoft Active Documents Since 
digital watermarks are used in digital systems, the simplest way to make a watermark active 
.s to mclude program code in it which can be executed by the computer system upon which 
rhe thgnal representation is currently resident. From the point of view of function, the code 
may be m any language for which the computer system can execute code. Practical 
however, the code is best written in a language such as Java- or Perl for whjch mQSt 
computer systems have interpreters. 

FIG. 6 is an overview of a system 601 for making an active watermark 619. The watermark 
•s made from watermark information 603. which contains owner information 605 access 
•nformanon 607, and owner-defined information 609 as before, but additionally contains code 
611. Code 61 1 may be standard code for a g.ven class of digital representations, or it mav 
be defined specifically for a given digital representation. Code 6 1 1 may of course also use the 
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other information in watermark information 603 as data. Watermark information 603 and 
digital representation 613 are input into watermark maker 615, which outputs digital 
representation 617, which is digital representation 6 1 3 modified to include watermark 619 
made from watermark information 603. Since watermark information 603 includes code 611, 
watermark 619 is an active watermark. 

FIG. 1 1 shows a preferred embodiment of access information 607. It includes fields as 
follows: 

an 8-bit permission (P) field which indicates the kind of access the user may have: 
among the kinds are access which permits display, access which permits storing a 
local copy, and access which permits printing. 

a four-bit sensitivity field whose value indicates the degree of sensitivity of the 
contents of the digital representation; 

a 32-bit allowed location field which contains the IP address at which the digital 
representation is permitted to be located; 

A 32-bit allowed period field which contains a period of time for which use of the 
digital representation has been permitted; and 

FIG. 7 is an example of a program which might be found in code 611. Program 70 1 is written 
in the Java programming language. It is then compiled into Java bytecodes which are 
interpreted by a Java interpreter. These bytecodes are included in the digital watermark. 
When program 701 is executed, a message indicating that digital representation 617 
containing the active watermark has been displayed is sent via the Internet to a system that 
has been set up to monitor the display of digital representation 617, perhaps for the purpose 
of computing license fees. Line 703 of the code sets up a socket s by means of which _ 
datagram may be sent to the monitoring system. Line 709 of the code finds the current 
Internet address a of the monitoring system which is specified at 705 by the name 
tsyscop.crg.edu.. Line 1715 makes a new datagram packet for the message; it 
•ncludes the message content, «XYZ Displayed, and the internet address a. Line 1719 
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finally, uses the send operation associated with the socket s to send the message, which the 
Internet will deliver to the destination specified by a. 

FIG. 8 shows a system 801 for executing the code in active watermark 619 Digital 
representation 6 1 7 containing active watermark 619 is input to watermark reader 803. wh>ch 
extracts watermark information 603 from active watermark 6 1 9 Info 603 includes code 611, 
which watermark reader 803 provides to code interpreter 805. Code interpreter 805 
interprets code 61 1 to provide instructions which are executable by the computer system upon 
which code interpreter 805 is running. In some embodiments, code interpreter is an 
interpreter provided by the computer system for a standard language such as Java; in others, 
interpreter 805 may be provided as a component of watermark reader 803. In such 
embodiments, code 611 may be written in a language specifically designed for active 
waiermarks. 

An active watermark 619 can cause the computer system in which the active watermark is 
read to perform any action which can be described by the code contained in the active 
watermark. The only limitations are those imposed by the fact that the code is pan of a 
watermark. One of these limitations is code size: code contained in a watermark must 
necessarily be relatively short; this limitation can be alleviated by compressing the code using 
a "non-lossy" compression technique, that is, one which does not result in the loss of 
information. Another of the limitations is that damage to the watermark may result in 
damage to the code; consequently, active watermarks may not work well in situations where 
the digital representation 617 is involved in "lossy" manipulations, i.e.. manipulations that 
cause loss of information in digital representation 6 1 7. Examples of such lossy manipulations 
are editing the digital representation, lossy translation of the digital representation from one 
format into another, lossy compression of the digital representation, and producing a new 
digital representation from an analog form made from an old digital representation (as would 
be the case, for example, if the code were obtained by reading the watermark from a paper 
copy of a document). 
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Of course, techniques like those discussed above with reference to exact matches of digests 
can be applied to recover code from a damaged watermark or from an analog form, and to 
the extent that such techniques are successful, active watermarks can be used even where 
lossy manipulations have taken place. For example, the watermark of an analog form may 
contam not only authentication information, but also code. If a copy machine contained a 
watermark reader and an interpreter for the code used in the active watermark the active 
watermark could be used, for instance, to prevent the copy machine from copying the anaios 
form. 



10 Ajnong the th.ngs that can be done with active watermarks are the following: 

Customizing the manner in which the digital representation containing the watermark 
is treated. Code 61 1 may differ for classes of digital representations, or may even be 
particular to a single digital representation; 

Having a digital representation send a message whenever it is displaved. copied 
printed, or edited; for example, whenever a document with an active watermark 
stored on a Web server is downloaded from the server, the active watermark can 
cause a message containing billing information to be sent to a billing server. 
Having the digital representation obtain locally-available information, which will then 
govern the behavior and usage of the digital representation; 
20 • Having a digital representation take protective action when a user tries to do 

something with it that is not permitted by access informarion 603; the protective 
action can range from a warning through sending a message or blocking the intended 
action to destroying the digital representation that contains the watermark. 

25 Watermark agents 

Digital representations pose special problems for their owners because, like all diaital data 
they can be easily copied and distributed across a network. These properties of diaita. data 
however, also make it possible to automate monitoring of the distribution and use of 
watermarked digital representations. One way to do this is the watermark spider As 
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mentioned in the Description of ike Prior Art, the watermark spider follows URLs to Web 
pages, wh.ch it retrieves and inspects for watermarks. If it finds one that is of interest it 
reports us findings to a monitoring program. There are two problems with the watermark 
sp.der: the first is that it is limited to digital representations which are access,b.e by URLs that 
are ava.lab.e to the pub.ic. Thus, the watermark spider wou.d not be able to locate a copv 
of a d lg :ta] representation on a WW client, as opposed to on a WWW server The other 
problem is that the spider must fetch each digital representation to be examined across the 
network. S.nce digital representations are often large, the need to do this adds substantia.lv 
to the volume of network traffic. 

Both of these problems can be solved by means of a network watermark agent that is a 
watermark monitor wh,ch uses the network to move from system to system where disiu, 
representations of interest might be stored. At each system, the watermark asent examines 
the system's file system for digital representations that have watermarks of interest If the 
watermark agent finds such a watermark, it may send a message with its finding via the 
network to a monitoring program. The watermark agent is thus able to mentor digital 
representations that are not available via pub.ic URLs and uses network bandwidth only 
relanvely rarely and only to send messages that are small in comparison with digital 
representations. In the following, the creation of a watermark agent and its behavior "„ a 
system will both be explained in detail. 

Creating a watermark agent: FIG. 9 

™„' !T a wa,enMrk raom,orins sys,em 901 wh,ch creatK and • « 

925 across a networic IM and fesponds ,„ ^ ^ 

" 7 aSeM 925 * * P "*™ ^ " » send itseif from on. node to another in 
««* .03. ,„ each nod , „ searcnes fcr wa[ermarked documems sends ^ ^ 

— g us findings „ monitoring system 90 , where messags hand|ef - ^ 

message, often by adding information ,o management data base 903. 
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Continuing in more detail, agent 925 has two main pans: agent code 927, which is executed 
when agent 925 reaches a node, and agent data 929. which contains information used by 
agent 925 m executing the code and in moving to the next node. At a minimum aeent code 
927 w„l mclude code which searches the node for files that may contam watermarks code 
that makes and sends any necessary messages to monitoring system 901, code that clones 
agent 925, and code that sends the cione on to the next node. As with the code in active 
watermarks, code 927 may be written in any language which can be executed m a node either 
standard languages such as Java or a specialized watermark agent language may be used. 

HG. 12 provides an example written in the Java language of code 1201 that a watermarkine 
agent 925 nught execute. Code 1201 searches the file system of the network node at which 
agent 925 » presently located for images files, checks each image file for a watermark if it 
finds a watermark, it performs the action required by the watermark and the node, and makes 
a message containing a list of the actions it performed. 

Continuing in more detail, code 1201 has two main sections, initialization 1203 and checking 
loop 1213. In initialization 1203, the first step is to instantiate a file filter to filter the files 
» the node's file system (1205). Then a function of the filter which locates image files is used 
to make a list filenames of the names of the image files in the file system (P07) 
Thereupon, ^formation about the environment of the node that the agent needs to check 
watermarks is retrieved and placed in a variable env (1209); finally, a data structure called 
results .s created to hold the results of the watermark checks 1211. 

In loop 1213, each file in filenames is examined in turn for a watermark (P15) if one 
■s found, the actions indicated at 12 17 are performed; first, the contents of the watermark are 
compared with the environment information to obtain a result called match (P19) Then 
match „ passed to a function which takes an action as determined by the value of match 
and returns a value result which represents the result of the action (1221)- finally 
result „ added to the data structure results (1223). then, at 1225 results is 
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returned. Depending on how the watermark agent is being used, results can then be sent 
in a message to monitoring system 90 1 . 

Continuing in more detail with agent data 929, agent data 929 includes a map 93 1. digital 
representation description 933. keys 934, and parameters 921. Map 93 1 is a list of addresses 
in network 103. Each address specifies an entity in network 103 that can provide an 
environment in which agent 925 can operate. The address may for example be an E-mail 
address or an IP address. Digital representation description 933 may be any information that 
describes the digital representations the agent is looking for. There may be a filter for the file 
names and there may also be identification information from the watermark. For example, 
if the files to be examined are . bmp files, the filter might specify * . bmp, indicating thai 
all files with the . bmp suffix are to be examined. If a watermark key is needed to read the 
watermark, keys 934 will contain that key and if the messages sent to monitor system 901 are 
to be encrypted, keys 934 will contain the key to be used in encrypting the messages. .Any 
available technique may be used to keep the keys secure. In a preferred embodiment, the 
parameters include 

the email address for messages sent by the agent; 
whether to report on files to which agent 925 had no access; 
date of last monitoring and whether to check only files updated since that date; 
whether to execute code 61 1 in an active watermark 619; and 
termination conditions for agent 925. 

Agent 925 is produced by agent generator 923, which can be implemented as a component 
of digital representation manager 131. Agent generator 923 makes agent 925 from 
information in management data base 903 and agent parameters 921, which here are shown 
being provided interactively by a user of monitoring system 901, but may also be stored in 
management data base 903. The information in management data base 903 includes a S em 
template 905(i), which is one of a number of templates that are used together with parameters 
921 and other information in management data base 903 to generate agent code 927 for 
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different kinds of agents 925. Suspicious sites 907 is a list of network locations which might 
be worth examining. One source of information for sites that should be on the list of 
suspicious sites 907 is of course messages received from previously-dispatched agents. 
Network information 909 is information about the network. Suspicious sites 907 and 
network information 909 are used together to make map 931 in agent 925. Digital 
representation information 91 1, finally, contains information about the digital representations 
that the agent will be looking for. The information is used to make DR Description 933. 
Information 91 l(i) for a given digital representation or group of digital representations may 
include a watermark key 913 for the digital representation's watermark and information from 
the watermark including owner ID 915, user ID 917, and permitted use information 919 
User ID 917 is an identification for the user to whom the digital representation was 
downloaded. Once agent 905 has thus been created by monitor system 901, agent 925 clones 
itself, makes the clone into the kind of message required for the first entity specified in map 
931, and sends the message to the first entity. Thereupon, agent 925 terminates itself 

Watermark agents in network nodes: FIG. 10 

FIG. 10 shows those components of a network node 1001 which are involved in the 
monitoring of the node by a watermark agent 925 The components include: 

agent engine 1003, which provides the environment in which agent 925 executes its 

code and which is the entity to which the message containing agent 925 is addressed; 

file storage 103 1, which contains the digital representations 1 023 that are of interest 

to agent 925; 

file system 1029, which makes the digital representations 1023 accessible as files; 
watermark reader 1019, which reads the watermarks; and 

code interpreter 1 0 11 , which interprets code in agent 925 and may also interpret the 
code in active watermarks, if that code is written in the same language as the code 
used in agent 925. 

SC 1035 is an optional secure coprocessor whose functions will be explained in more detail 
in the discussion of security. 
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Operation of components 1001 is as follows: When the message containing agent 925 arrives 
in agent engine 1003 from network 103. agent engine 1003 extracts agent 925 from the 
message and, at a convenient time, uses code interpreter 101 1 to begin executing its code. 
What the code does is of course arbitrary. Typically, it will do the following: 
I ■ Send a message to system 90 1 indicating its arrival in the node; 

2. Obtain the file filter from DRDESC 993 and give it to spider 1009 to make a list of 
files that match the filter; 

3. For each file on the list, do the following: 

a. use spider 1009 to get the file ID for the.file; 

b. give file ID1021 to watermark reader 1019. which uses the watermark key 
from keys 934 to read the watermark, if any; 

c receive the watermark content 1017; 

d. process watermark content 1017 as specified in code 927. Actions might 
include sending a message to system 901 or passing the code and data 1015 
from an active watermark to code interpreter 1011 for execution and 
receiving data 1 0 1 3 in return; 
4. When all of the files have been processed, 

sending a message to monitor system 901 with summary information about 
the results of the visit and the next node to be visited; 
b. making a clone of agent 925 and sending the clone to the next address 

specified in map 93 1 ; and 
c terminating agent 925. 
As previously indicated, what a watermark agent can do is essentially arbitrary If the 
documents being dealt with by the watermark agent have active watermarks, there are anv 
number of ways of dividing the work of processing of documents of interest between the 
code ,„ the watermark agent and the code in the active watermark. For instance in the 
example above, step 3(d) above could consist simply of executing the code in the document's 
active watermark. 
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The actions performed in step3(d) will typically be performed when 

the information in the watermark does not match the time or place where agent 925 found the 
file or the time and/or place are inappropriate for the file's access privileges. The action may 
be one of a pre-defined set specified by parameters in parameters 92 1 , it may be one defined 
by agent 925's code 927, or it may be one defined by an active watermark. Among the pre- 
defined actions are: 

1 Destroy the file if the file's sensitivity level is very high; 

2. Remove the file to a safe place if the sensitivity level is medium; 

3. If the sensitivity level is low, 

a. Warn the local administrator or webmaster of the violation if the sensitivity is low; 

b. Warn the recipient of the violation if the sensitivity is low; or 

c Send a message to the file's owner reporting the violation if the sensitivitv is low 

4. If the sensitivity level is very low, send a message to monitor 901 without disturbing the 
local host and local administrator . 

Before going on to the next destination, watermark agent 925 may, wait for a message from 
momtor 901 containing information about the next destination; the information may 
include: 

• The time of the last visit by an agent to the destination; 

• Information about the destination, for example detailed information about the digital 
representations to be examined there. 

Nontraveling watermark agents 

An important difference between a watermark agent and a watermark spider is that the 
watermark agent interacts with the document in the system where the document is being 
stored or processed, and can thus perform far more functions than a watermark spider can 
A further consequence of this difference is that a watermark agent need not travel, but can 
amply be incorporated as a permanent component of a system. For example, a copier cou.d 
■nclude a watermark agent that read the watermarks of paper documents being copied and 
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prevented the copier from copying a document when its watermark indicated that the 
document was not to be copied. .An important application of such a non-traveling 
watermark agent would be to prevent the copying of paper digital cash. 

Of course, if the copier had access to a network, even the "non-traveling" watermark agent 
could at least travel via the network to the copier, and the network would provide a 
convenient way of updating the copier's watermark agent. "Non-traveling" watermark 
agents could of course be distributed in a similar fashion to any system access.ble via the 

network. 



Security considerations 

In some cases, for example in private military or business networks or systems, aaent <m may 
not operate in a hostile environment, and monitor 901 and agent engine 1003 mav even be 
•mplememed as integral pans of the operating system. In most cases, however, agent' 925 will 
be operating in an environment which is hostile in at least four respects: 

The node to which agent 925 sends itself is properly suspicious of messages from 

outside that contain code to be executed on the node; 

to the extent that users on the node have violated the conditions under which thev 
receded a digital representation, they will want to hide their behavior and/or disable 
agent 925; 

users on the node may want access to the keys and other data carried bv aeent 9">5- 
and ' ~ 

other users of network 103 may be interested in the content of the messages being 
exchanged between agent 925 and monitor 90 1 . 

The first of these problems is the "ma.icious agent problem". It is genera, to svstems that 
download and execute code, and the same solutions that are used in those cases can be 
apphed to agent engine .003 and agent 925. For example, if the watermark agenfs code is 
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wmten ,n Java, the system on which it is nin will have whatever protections are provided by 
the Java .nterpreter. If managers of nodes are reasonably certain that agent engine 1003 and 
agents 925 will not do any damage to the nodes, they can be made to accept engine 1003 and 
agents 925 s.mply as . condition of downloading digital representations. For example the 
transacts by which a digital representation manager downloads a digital representation to 
a node mjght mclude a message to agent engine 1003 confirming the existence and operabilitv 
of agent eng,ne 1003. If the message were not properly answered, the digital representation 
manager m.ght require that the node download and install agent engine .003 before 
proceeding further with the transaction. 

The remainder of these prob.ems are termed "malicious node problems". They can be solved 
by standard cryptographic techniques, as described in Schneier. supra. For example the 
d.g!tal representation manager and each agent engine 1003 might have a public kev-pnvate 
key P a,r. In that case, network information 909 would include the public key for the agent 
engine 1003 at a given node and the pub.ic keys for the agent engines 1003 in the nodes to 
be v ls ,ted would be inc.uded in map 93! . Any message sent by the digital representauon 
manager or by an agent 925 to an agent engine 1003 can be encrypted using agent engine 
100, s pubhc key and any message sent by an agent engine 1003 or an agent 925 to a digital 
represents manager can be sent using the digital representation manager's public kev The 
pubhc key for the digital representation manager can of course be included in aeent 9^5's 
keys 934. Authentication of messages can be done using standard digital stature 
techmques; for example, agent data 929 might include a digital signature from thecal 
representation manager for agent 925. messages from the digital representation manager to 
agent engine 1003 can include the digital represents manager's digital signature and 
messages from agent engine 1003 can include agent engine 1003's digital signature 

If the watermarks are made using encryption techniques, as described in E. Koch and J Zhao 
Towards Robust and Hidden Image Copyright LabeHng", ^ the agent must havfi . _ 
of decrypts the watermark. Depending on the situation, the watermark may be encrvpted 
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with the watermark agent's public key and authenticated with a digital si e nature in the same 
fash.cn as other messages to the agent engine or the watermark may have its own kev 9 i 3 
In the former case, the, the watermark agent's private key must be protected and in the latter 
watermark key 913 must be protected, since access to the key wou.d perm„ those intent on 
steahng d.gital representations to remove or alter the digital representation's watermark 
WMe agent 925 is in transit, watermark key 9,3 can be protected by encryption in the same 
fas hl on as the rest of the information in agent 925; once agent 925 has been decrypted 
watermark key 913 and agent engine 1003's private key must be protected in the node 
currently bring visited by agent 925. Agent engine 1.003's private key must further be 
protected to prevent a user of the node currently being visited by agent 925 from using the 
pnvate key to decrypt messages addressed to agent engine 1003 or append agent 1003's 
digital signature. 

One way of solving these key protection problems is a secure coprocessor, as described in 
"X Tyger and Bennet Lee, Secure Coprocessors in Electronic Commerce Applications 
HRST USENDC WORKSHOP ON ELECTRONIC COMMERCE, JULY 1995 As shown 
« 1033. a secure coprocessor includes secure storage 1035 and a secure processor ,045 
Secure storage 1035 may only be accessed via secure processor 1045. and secure coprocessor 
1033 ,s built in such a fashion that any attempt to access the information in secure 
coprocessor ,033 other than via secure processor ,045 results in the destrucuon of the 
-formanon. Secure coprocessor ,033 is able to write information to and read information 
from secure storage ,035 and also does encryption and decryption and makes and verifies 
dqptd matures. These operations may be done entire.y by executing code stored in secure 
storage ,035 or by means of a combination of code and specialized hardware devices as 
shown at ,047 and 1049. The keys used in encryption, decryption, and in making digital 
statures and verifying them are stored in secure storage ,035. Shown in FIG 10 are 
WMkey 9,3 f or the watermark, monitor public key ,039. agent engine public key ,04, and 
agent engme private key ,043. In the case of the public keys, storage in secure storage ',035 
■s s,mp,y a matter of convenience, and secure processor ,045 may provide access to the 
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public keys in response to requests from components of node 1001; in the case of WMkev 
913 and agent engine 1003's pnvate key 1043, the decrypted keys 913 and 1043 are used only 
within secure processor 1033. 

In the context of system 1001, when a message encrypted with agent eneine ,003's public kev 
1041 amves in agent engine 1003, agent engine 1003 uses secure processor 1033 to decrypt 
the message; if the message contains an agent 925, agent engine 1003 also uses secure 
processor 1033 to verify that agent 925's digital signature is from the digital representat.on 
manager and to decrypt WMkey 913. The decrypted key is not returned to aeent engine 
1003, but „ stored in secure storage 1035. SWM reader 1019 then uses secure coprocessor 
10, , to decrypt the watermark in the digital representation currently being checked by agent 

Applications using watermark agents 

Since a watermark agent is programmed, it can do literally anything. The flexibility of 
watermark agents is increased when their use is combined with that of active watermarks 
One set of applications for watermark agents is monitoring the use of copyrighted digital 
representations for the copynght owner or a licensing agency. A copyright owner or 
1'cens.ng agency, for example, may use watermark agents to locate unlicensed copies of 
d-gnal representations or to periodically monitor the use of licensed copies. A document with 
an acnve watermark could increment a usage count maintained in agent engine 1003 for a 
node each time it was printed and agent 925 could read the count on its visit to the node 
report the current count value back to management database 903, and reset the counter. ' 

Another set of applications is monuoring the use of digital representations to avoid liabi.itv 
for mrnngement. For example, a corporation might want to be sure that it has no 
unauthonzed digital representations in its network a,d that the authorized ones are being used 
» accordance with their .icense terms. The agent can monitor the use of the digital 
representations in the corporate network in the same fashion as it does for the .icensing 



31 



WO 99/57885 



PCT/EP99/02928 



10 



15 



20 



25 



agency. In this instance, the monitoring might even include destroying illegal copies. 

Yet another set of applications is preventing unauthorized copying, scanning, or printing. 
This can be done by means of "nontraveling" watermark agents on servers and clients in the 
network or even by means of "nontraveling" watermark agents built into devices such as 
copiers, scanners, or printers. For example, if a "No copy" watermark is embedded in 
currency and a photocopier has an agent that looks for such a watermark and inhibits copying 
when it finds the watermark, the photocopier will not make copies of currency. 

Watermark agents can also be used to enforce military or corporate document security rules. 
In such an application, the document's security classification would be embedded in it as a 
watermark and the watermark agent would search the military or corporate file systems and 
networks for documents that were not being dealt with as required by their security 
classification. Examples would be documents that were in the wrong place or had been kept 
longer than a predetermined period. Actions taken by the agent can range from repons and 
warnings through changing the access rights to the document or moving the document to a 
safe location to immediate destruction of the out-of-place document. Again, the agent that 
does this need not travel, but may simply be a permanent component of the file system. 

Watermark agents, finally, can be used to find lost documents in military or business file 
systems or networks. If each document has a unique identifier associated with it and that 
■dentifier is on the one hand kept in a database and on the other hand incorporated into a 
watermark in the document, a watermark agent can simply be given the universal identifier 
and sent to search the file system or network for the document. Once the agent has found it. 
it can report its location to whomever sent the agent out. 

Conclusion 

Tne foregoing Detailed Descnpnon has disclosed to those skilled in the relevant arts how to 
make and use documents with authentication that withstands conversion between an analog 
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torm and a digital representation of the document, how to make and use digital 
representations with active watermarks, and how to make and use watermark a-ents 
■nciudrng mobile watermark agents, and has further disclosed the best mode presently known 
to the inventors for making such authentications, making active watermarks, and making 
watermark agent, The disc.osed techniques are exceedingly general and may be implemented 
m many d.fferent ways for many different purposes. For examp.e, the authentication 
techniques may be based on any kind of semantic information and there are many wavs of 
denvmg the authentication information from the semantic information, niacins the 
authenncauon information in the digital representation or the analog form, and comparing the 
authent.cation information. Similarly, the program code for an active watermark mav be 
written «n any programming language, may be in source or object form, and may when 
executed, perform arbitrary operations. Watermark agents, too. may perform arbitrarv 
actons and employ various techniques for sending messages and traveling from node to node 
«n a network. The watermark agents can of course perform authentication information and 
can execute code from active watermarks. 

Since the techniques are so general and may be implemented in any number of wavs the 
Defied Description is to be regarded as being in all respects exemplary and not restrictive 
and the breadth of the invention disc.osed herein is to be determined not from the Derailed 
Descry but rather from the claims as interpreted with the full breadth permitted bv the 
patent laws. 
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What is claimed is: 

1. Apparatus which adds authentication information to a digital representation, 
the apparatus comprising; 

an authenticator which uses semantic information in the digital representation to 
produce the authentication information, the authentication information being preserved in an 
analog form produced from the digital representation; and 

an incorporator which incorporates the authentication information into the digital 
representation such that the meaning of the semantic information is not altered thereby 

2. The apparatus set forth in claim 1 wherein: 

the incorporator incorporates the authentication information in a form which cannot 
be perceived by unaided observation of the analog form. 

3. The apparatus set forth in claim 2 wherein: 

the form which cannot be perceived is a digital watermark. 

4. The apparatus set forth in claim 1 wherein: 

the incorporator incorporates the authentication information into the digital 
represent*™ in a form which can be perceived by unaided observation of the analog form. 

5. The apparatus set forth in claim 4 wherein: 

the form which can be perceived is a bar code. 

6. The apparatus set forth in any one of claims 1 through 4 wherein: 

the authentication information is a digest made from the semantic information. 

7. The apparatus set forth in any one of claims 1 through 4 wherein: 

the authentication information is robust with regard to insubstantial errors in reading 
the semantic information from the analog form. 
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8. The apparatus set forth in claim 7 wherein: 



the authentication information reflects at least in pan an order of the semantic 
3 information. 

1 9. The apparatus set forth in any one of claims 1 through 5 wherein: 

2 the digital representation is a digital representation of a document; and 

the semantic information includes alphanumeric characters in the document. 



10. The apparatus set forth in claim 9 wherein: 

the digital representation is a digital image of the document; and 
the apparatus includes an optical character recognizer that recognizes the 
4 alphanumeric characters. 



1 11. The apparatus set forth in claim 10 wherein: 

2 the digital image is produced by scanning an analog form of the document, 

3 whereby the authentication information may be added to a copy made from the scanned 

4 analog form. 



12. The apparatus set forth in claim 9 wherein: 



the digital representation includes digital codes representing the alphanumeric 
3 characters; and 



the apparatus includes a parser that recognizes the digital codes represent^ the 
5 alphanumeric characters. 



13. The apparatus set forth in claim 9 wherein: 

the analog form produced from the digital representation is paper digital cash. 

14. The apparatus set forth in claim 9 wherein: 
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the analog form produced from the digital representation is a paper digital check. 

15. The apparatus set forth in claim 9 wherein: 

the analog form produced from the digital representation is an identification card. 

16. A printer characterized in that: 

the printer employs the apparatus set forth in claim 9 to add the authentication 
information to an analog form printed by the printer. 

17. Apparatus for determining authenticity of an analog form, the analog form containing 
first authentication information that is produced using semantic information in the analog 
form and is incorporated into the analog form such that the meaning of the semantic 
information is not altered thereby, 

the apparatus comprising: 

a semantic information recognizer that recognizes the semantic information in the 
analog form; 

an authentication information reader that reads the first authentication information 
from the analog form; and 

an authenticator that computes second authentication information from the recognized 
semantic information and determines whether the analog form is authentic by comparing the 
first authentication information with the second authentication information. 

18. The apparatus set forth in claim 17 wherein: 

the authentication information is incorporated in a form which cannot be perceived 
by unaided observation of the analog form. 

19. The apparatus set forth in claim 18 wherein: 

the form is a digital watermark; and 

the authentication information reader is a digital watermark reader. 
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20. The apparatus set forth in claim 17 wherein: 

the authentication information is incorporated in a form which can be perceived by 
unaided observation of the analog form. 

21. The apparatus set forth in claim 20 wherein: 

The form is a bar code; and 

the authentication information reader is a bar code reader. 

22. The apparatus set forth in any one of claims 17 through 21 wherein: 

the authentication information is a digest made from the semantic information. 

23. The apparatus set forth in any of claims 17 through 21 wherein: 

the authenticator computes the second authentication information in a fashion which 
is robust with regard to insubstantial errors made by the semantic information recognizer 

24. The apparatus set forth in claim 23 wherein: 

the authenticator compares the first authentication information with the second 
authentication information such that a partial match within a threshold indicates that the 
analog form is authentic. 

25. The apparatus set forth in claim 23 wherein: 

the authentication information reflects at least in pan an order of the semantic information. 

26. The apparatus set forth in any one of claims 17 through 21 wherein: 

the authenticator compares the first authentication information with the second 
authentication information in a manner which is robust with regard to insubstantial errors 
made by the semantic information recognizer. 



WO 99/57885 PCT/EP99/02928 

27. The apparatus set forth in claim 26 wherein: 

the semantic information is subject to constraints; arid 

the authenticator includes an error corrector that employs the constraints to correct 
errors in the recognized semantic information and uses the corrected recognized semantic 
information to recompute the second authentication information when there is not a precise 
match between the first authentication information and the second authentication information. 

28. The apparatus set forth in any one of claims 17 through 21 wherein: 

the analog form is produced from a digital representation that includes the first 
authentication information. 

29. The apparatus set forth in any of claims 17 through 21 wherein: 

the analog form is a document; 

the semantic information includes alphanumeric characters in the document; and 
the semantic information recognizer is an optical character recognizer. 

30. The apparatus set forth in claim 29 wherein: 

the document is paper digital cash. 

31. The apparatus set forth in claim 29 wherein: 

the document is a paper digital check. 

32. The apparatus set forth in claim 29 wherein: 

the document is an identification card. 

33. A scanner characterized in that: 

the scanner employs the apparatus set forth in claim 29 to determine authenticity of 
analog forms scanned by the scanner. 
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34. A method of adding authentication information to a digital representation, the method 
comprising the steps of: 

producing authentication information from semantic information in the digital 
representation, the authentic information having the property that it is preserved in an analog 
5 form produced from the digital representation; and 

incorporating the authentication information into the digital representation such that 
the meaning of the semantic information is not altered thereby. 



35. A method of determining authenticity of an analog form, the analog form containing first 
authentication information that is produced using semantic information in the analog form and 
is incorporated into the analog form such that the meaning of the semantic information is not 

4 altered thereby, 

5 the method comprising the steps of: 
recognizing the semantic information in the analog form; 
reading the first authentication information; 

computing second authentication information from the recognized semantic 
9 information; and 

10 determining whether the analog form is authentic by comparing the first authentication 

1 1 information with the second authentication information. 



1 36. An authenticated analog form comprising: 

semantic information and 

authentication information which is pan of the analog form but does not substantially 
affect the meaning of the semantic information, the authentication information being produced 
5 using the semantic information. 
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37. An authenticated digital representation comprising: 

a digital representation of semantic information; and 

authentication information, the authentication information being pan of the digital 
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representation but not altering the meaning of the semantic information, being produced using 
the digital representation of the semantic information, and surviving in an analog form 
produced from the digital representation. 



40 



WO 99/57885 



PCT/EP99/02928 



1/11 




WO 99/57885 



PCT/EP99/02928 



2/11 



IIIIIH! 



SEMANTIC 
DIGEST, 207 



acrn^pxev^ OK a v ^P ( P'n70pYnvPopi£r|K^p 
oriTvpopriYX, pir|(pYpiT(p7u\)pvmpTipv7ap7pvv 
7acopYn KpV7a5T177C4)Vp7CieTr| p7iir|e7tPvK4»v {topi 
eynapveiT|7naK^y7cpiYK:av;aeKn(t)9<l)0cai)7rt 
9p;a(|)K<p7tpiCGYn9;Ka^ 

oi«Kcni7cpa(X)r|q)77C9K^vPma)apYK:<Tvp7ccoipKa 

vpaicv7CCOip7CcyvPaK(t)iipeipYnvP^vG3|ir|cpY7icom 

Tt>p7ti\)VC5?icor|p^^(pp|i;oTp(pr|;8v|ap;8K(p;8Kvp 

<t>K(pYn!^M%vtOpK^o^^ 

pip\)Y^P^9cop\)YnPM-;^M-^oexiY[[\)Yipvp,VT^8it) 

\|/q>8pA,v£Ti , o'nKp]LiKvp;A,a87r8oex\)i[\}/oi(t)PK|ip, 

9YErcxur|KvPrc£m>r|A|iP^ 

-ooco7ipteconeY.0M.Ti ;Xco\)priG5r| ;oco£p\)\yxr|r|t>r|Y; 

oei\|/o;ei\|/er|Yoei\)\(/eoTiYE;8o\|n)Y^9K7r|95T|X;8 

crn(pY;^ei^oe\)Ti j^STicptpKcpXKcpjxrira^Kri y,ii$usX 

(pt||ivp(pr|(>\)yTp£U)i^ 

a^eYETCoxi/ep^^^xvpMY^V^KWYW^pSeocoe 

px(|)G3xG3PPvTi(pM9m9T|Yi^pct>8£aco0aYn9OKi|LL 

vnYix\)\(/(pYVpjiG3VY9<t )>0 P r l ^vyh paxtxpYVYH <t>upoi 

woe8KxvK<pST|(|)\|/p\)epxvtnM(>T|<()\}/pY8pE^8G5a(t) 

8xeY8TE|Livp(pY9V^P^0p8YeT8iLivG3r|^(ppa)p^ 

(PYi^M^khmtuyH^PW^^ 
YjivvPri PpG5Y8xeG58(|)ETXp|i<pvr| PYn|/<t>pPG5c|>YrD 




SEMANTIC 
INFORMATION, 205 



ANALOG FORM, 203 



201 



Fig. 2 



WO 99/57885 



3/11 



PCT/EP99/02928 



WAA 
WITH 
SEMANTIC 
DIGEST, 307 



acTnSYpTevi/DOK^vGSpcpTiYOpYnvpopieriKXp 
oriYvpopriYA, piT|<pYpix(p7tn)pvnipT| pv7tipYpvv 
mcopynKpv7a5Ti7ic^)vp7ueTr|p7ar|e7cPvK(i)vp7cpi 
eyr| opveiri Y7taKVY7ipiYKav ;cc£kt| ^cp^OcmyyTCOcI) 
cpp;oc|)Kcp7tpiG)Yn9;Ka(^ 
oiWK<Tn^P^cori(p77C9K<)>vpTncoapYKavp7ccoipKa 
vpaKV7tcoip7cc7vPaK(t)TiP£ipYnvp|iva5^r|(p77ccom 
x\)pmt)V05Xcor|pY;?i(pp|i;oxp(pr|;8v|ip;8K(p;5Kvp 

pip\)Yicp7c9copt)Yr|pM-;^)ii7T:oeTiY[['UYipvp,vxX8i\) 
\y<p8pA,v£Ttt>r| Kpy,iw §; A,gSn5qexm (^o^pKjip, 
(fryE7tx\)T|KVprc£ixJo^ 

\)OC0Kp[£C07iEY.<()(j|T| ;A,co\)priG5r| ;oco£p\)\}/Tr|T|\pT|Y; 
0£i\|/o;£i\|mr|YO£i|\)y£or|YE^^ 
"^rrttpYAgvooe^Ti ;^8r|(p(pK(pXK(p(i.TiG3\Kr|Y s Hp03i 
(piliavpcpiTej^ 

a(()£YEXco\|/£pr|(t)9Iwp^YW^ K WY T V^p5e<7to£ 
pX((>GJx ro PP v ^w|p w 9^YW^ 
vtiy^WYvP^^W^P^vYH^ 
too£SKxvK9Sr|(|>\|irp\)£pxvG3^ 
Sx£Y8x£pA'p(pY9\|K>pxiT^ 
(pyivy,ii$K^]iTvyi\yvpw3 cpjipjl (|) p \)p jig v_Pjir| p 
\|/|iv v Ptj p pas Y8x£SS(|)£xxp p.<pvri pyni/<f>p Pgs^ytd 



SEMANTIC 
INFORMATION, 305 



ANALOG FORM, 303 



301 



Fig. 3 



WO 99/57885 



PCT/EP99/02928 



4/11 



DR 



SEMANTIC 



INFO 



L 



"205 



403 



SEMANTIC \ 



READER 



-405 



SEMANTICS 
DATA, 407 



DIGEST 
MAKER 



409 



SEMANTIC 
DIGEST, 411 



DIGEST 
INCORPORATOR 



413 



DR403 AND 
SEMANTIC DIGEST, 411 



ANALOG 

FORM 
PRODUCER 



415 




ANALOG FORM, 203 



Fig. 4 



WO 99/57885 



PCT/EP99/02928 



5/11 




ANALOG 
FORM, 503 



SEMANTIC 
DIGEST 
READER 



505 



SEMANTICS 
READER 



507 



OLD 

SEMANTIC 
DIGEST 515 



SEMANTICS 
DATA, 509 



SEMANTIC 
DIGEST 
MAKER 



511 



NEW 

SEMANTIC 
DIGEST, 513 



COMPARATOR 



517 



501 



COMPARISON 
RESULT, 519 



Fig. 5 



WO 99/57885 



PCT/EP99/02928 



6/11 



OWNER INFO, 605 



ACCESS INFO, 607 



OWNER - DEF. INFO. 609 



CODE, 611 



601 



WATERMARK 
INFO, 603 



DIGITAL 
REP, 613 



WATERMARK MAKER 



615 



! ACTIVE 




i WATERMARK 


1 619 



DIGITAL REP WITH 
WATERMARK, 617 

Fig. 6 



WO 99/57885 



PCT/EP99/02928 



7/11 



^703 

DatagramSocket s = new DatagramSocket (0);^ 

705 

Inet Address a = ^709 
ne t Address .get ByName ( "syscop. crcg.edu" 

DatagramPacket p = new DatagramPacket ( ■ XYZ 
Displayed", 13 , a, 14 1715 



s . send(p ) ; 1711 1713 
1719 

701 



Fig. 7 



WO 99/57885 



PCT/EP99/02928 



8/11 



ACTIVE 
WM 



619 



DIGITAL 
RER, 617 



INFO, 603 



CODE, 611 



l ! 



WATERMARK 
READER, 803 



CODE, 611 



CODE 
INTERPRETER 



801 



805 



EXECUTABLE 
INSTRUCTIONS, 807 



Fig. 8 



WO 99/57885 



PCT/EP99/02928 



9/11 



905(i) 



\ 





• 

• 


AGENT 


• 


ID 




• 

• 





SUSPICIOUS V 
SITES 



907 



MANAGEMENT DB. 903 



909 



J 



NETWORK 
INFO 



• 

• 






• 




I/ 913 

y-915 


WM KEY 


OWNER ID 


USER ID'S 


^917 


PERMITTED USE 
INFO 


^919 






• 




• 

• 



DR INFO, 911 (i) 



AGENT 
PARAMETERS 



V 



921 



AGENT 925 



901 



920 



J 



MESSAGE 
HANDLER 



AGENT 
GENERATOR 
(IN DRMG 131) 



923 



MAP, 931 


DR DESC, 933 


PARAMS, 921 

AGE 
CODE 


KEYS. 934 

NT 
,927 



} 



AGENT 
DATA, 929 




MESSAGES 
FROM AGENTS, 935 



Fig. 9 



WO 99/57885 



PCT/EP99/02928 



10/11 



103 



DATA, 
1013 



1011 




AGENT 

ENGINE, 

1003 



CODE 
INTERPRETER 



\ 



-1019 

RLE 
SYSTEM 
COM- 
sAAANDS., 
J025 



DRS, 1023 



FILE 
SYSTEM 
COM- 
MANDS., 
1027 



FILE SYSTEM 



l_ 



WM KEY, 913 



MON KEY, 1039 



SS, < 
1035 ^ 



_AEKU,_10_41_ 
AEKR, 1043 



PROC,^ 
1045 ^ 



' ENCR/DECR., 
i 1047 



DIG. SIG. 
VER.^1049 

SC, 1033 



1029 



DRS 102 3 



FILE STORAGE, 1031 



1001 



Fig. 10 



WO 99/57885 



PCT/EP99/02928 



11/11 





SENSITIVITY 1103 








P, 1101 


pi 




AL, 1105 


AR 1107 


UD, 1109 





607 

Fig. 11 



1205 



1207 



// Instantiate a file filter. 
FileFilter filePilter =new FileFilter ( ) ; 
// filter out all image files from the file system. 
String {} filenames = f ileFilter . f ilterlmages ( ) ; 
// get host environment information. 
Envlnfo env = getEnvInf o ( ) ; ~\--|209 

// construct a new vector used to store action results. 
Vector results = new VectorO;-^^^ 

for(int i=0; i<filenames. length; i++) { 

// check each image file for watermark. ^1215 
String watermark = checkWatermark ( filenames [i] ) ; 
if (watermark != null) { // if a watermark is found 
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// match the watermark with the host environment 
String match = mat chEnv (watermark, env) ; 
// take action according to the matching result. 
String result = takeAction (match) ; 
// add the result to the vector. 

results .addElement (result) ; A 

v 1223 



f 1213 



>1217 



-1221 



return results; 
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